Skip to content

ENH: Add schema migration support for versioned resources - #298

Merged
GibranAlfa merged 8 commits into
equinor:mainfrom
GibranAlfa:migration-manager
Aug 17, 2026
Merged

ENH: Add schema migration support for versioned resources#298
GibranAlfa merged 8 commits into
equinor:mainfrom
GibranAlfa:migration-manager

Conversation

@GibranAlfa

@GibranAlfa GibranAlfa commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Resolves #249
Resolves #305

Most lines are documentation and tests.

Add automatic, forward-only schema migrations for project config, user config, and internal mappings.

  • Load: Old data is migrated in memory. Loading does not change the stored file, create a cache revision, or add a changelog entry.
  • Save: When the resource is next saved, the original JSON is preserved as a normal cache revision before the migrated data is written.
  • Cache: Existing cache retention rules apply. Old cache revisions are migrated in memory when read.
  • Restore: Old cache revisions are migrated to the current schema before they are restored. Restore keeps the existing changelog behavior.
  • User experience: Migration happens automatically. Users do not need a separate endpoint or migration action.
  • Added a backup for resources that require migration. The exact original JSON is saved under .fmu/migration-backups/, with timestamp, model name, and source schema version in the filename.. Backup failures are logged but do not block the save. Users can copy a backup manually when rolling back to an older release.
  • Changed default cache_max_revisions from 5 to 10.

This also adds maintainer documentation and exports MigrationError from the public package API.

Related issue in fmu-settings-api is made to update the API to catch MigrationError properly.

Checklist

  • Tests added (if not, comment why)
  • Test coverage equal or up from main (run pytest with --cov=src/ --cov-report term-missing)
  • If not squash merging, every commit passes tests
  • Appropriate commit prefix and precise commit message used
  • All debug prints and unnecessary comments removed
  • Docstrings are correct and updated
  • Documentation is updated, if necessary
  • Latest main rebased/merged into branch
  • Added comments on this PR where appropriate to help reviewers
  • Moved issue status on project board
  • Checked the boxes in this checklist ✅

@GibranAlfa GibranAlfa self-assigned this Jul 29, 2026
@GibranAlfa GibranAlfa added the enhancement New feature or request label Jul 29, 2026
Comment thread src/fmu/settings/_migrations/manager.py
@GibranAlfa
GibranAlfa requested a review from slangeveld July 29, 2026 10:43
@GibranAlfa
GibranAlfa force-pushed the migration-manager branch 5 times, most recently from 25aa84c to d874d87 Compare July 31, 2026 14:20
@GibranAlfa
GibranAlfa requested a review from tnatt July 31, 2026 15:48

@slangeveld slangeveld left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work this and very nice that you have documented the migration process so well in the new README! A few comments and questions to discuss mainly on namings and structuring

Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_resources/cache_manager.py Outdated
Comment thread src/fmu/settings/_resources/cache_manager.py
Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_migrations/README.md Outdated
@GibranAlfa

GibranAlfa commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Added a backup for resources that require migration. The exact original JSON is saved under .fmu/migration-backups/, with timestamp, model name, and source schema version in the filename. Backup failures are logged but do not block the save. Users can copy a backup manually when rolling back to an older release.

Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_resources/pydantic_resource_manager.py Outdated
Comment thread src/fmu/settings/_migrations/README.md Outdated
Comment thread src/fmu/settings/_migrations/manager.py
@slangeveld

Copy link
Copy Markdown
Collaborator

Since you change the behavior of the cache (changing max revisions from 5 to 10) in this PR, can you create an issue explaining why this has been done and reference it in the PR description?

Then we have it documented for later why we did this.

@GibranAlfa

Copy link
Copy Markdown
Collaborator Author

Yes, I just created the issue and resolved it in the PR desc.

@slangeveld slangeveld left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there 💪

Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/manager.py Outdated
Comment thread src/fmu/settings/_migrations/README.md
Comment thread src/fmu/settings/_migrations/README.md
Comment thread src/fmu/settings/_resources/pydantic_resource_manager.py Outdated
Comment thread src/fmu/settings/_resources/pydantic_resource_manager.py Outdated
Comment thread src/fmu/settings/_resources/pydantic_resource_manager.py Outdated
Comment thread ARCHITECTURE.md Outdated
Comment thread ARCHITECTURE.md Outdated
Comment thread src/fmu/settings/_migrations/README.md
Comment thread src/fmu/settings/_migrations/README.md
Comment thread tests/test_migrations/test_migration_manager.py Outdated
Comment thread tests/test_migrations/test_migration_manager.py
Comment thread tests/test_migrations/test_migration_manager.py
Comment thread tests/test_migrations/test_resource_migration.py Outdated
Comment thread tests/test_migrations/test_resource_migration.py Outdated
Comment thread tests/test_migrations/test_resource_migration.py Outdated
Comment thread tests/test_migrations/test_resource_migration.py Outdated
Comment thread tests/test_migrations/test_resource_migration.py
Comment thread src/fmu/settings/_resources/pydantic_resource_manager.py Outdated

@slangeveld slangeveld left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Thanks for addressing all the comments

@GibranAlfa
GibranAlfa merged commit 8a5cb5b into equinor:main Aug 17, 2026
9 checks passed
@GibranAlfa
GibranAlfa deleted the migration-manager branch August 17, 2026 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change default cache revisions from 5 to 10 Add migration manager to handle migration of resources in .fmu

2 participants